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Abstract 

Many approaches are usad to measure quality ir*: graduate education. 
In this study, the literature related to four approaches is rev^'ewed: 
measuring quality by reputation, by scholarly productivity, by correlating 
reputation and scholarly productivity, and by multiple measures. This study 
represents a multiple measures approach to assessing quality arid describes 
and analyzes internal quality indicators for doctoral examinations in the 
Graduate School at Ohio State University. Graduate School Representatives 
judge doctoral examinations and report their judgments to the Dean of the 
Graduate School. The judgments are tabulated and presented as quantitative 
and qualitative indicators of quality. Row these judgments are converted 
to useful data and how these data are presented and used are emphasized. 




Blackburn and bingenfelter (1973) have said that the consideration of 
quality in any secting provokes controversy; Their remark is especially 
pertinent to the assessment of quality in higher education: The Eartter 
Reports (1966, 1977), for example, drew a flurry of negative reaction. 
Breneman (Bptions, 1977), for one, objected to the Inherent subjectivity of 
the studies and called them codified gossip, giving eurj^ent meaning to 
Dr. Johnson's obrervation that "a compendium of gossip is still gossip." 
More recently, a request to participate in a reputational study compelled 
on^ faculty member to write: 

... I urge that all those who are sent this questionnaire, 
and all others who have interest in the quality of education, 
resist being used by professional pollsters as pawns in 
superficial surveys (Rice, 1980, p. 77). 
Studies attempting to assess quality in graduate education have been a 
matter of controversy ever since the first one was publisher in 1925 
(Hughes, 1925); apparently, they continue to be. 

Despite the diversity of opinion over the methods, the uses, and the 
abuses of studies about quality, research in this area continues. It is 
not surprising then to learn that the Conference Board of the Associated 
Research Councils (1978) has planned and is now conducting a national study 
of quality in research doctorate programs. No doubt there will be others. 
This paper concerns the assessment of quality at the program level in a 
single institution and limits its focUs to doctoral programs. Therefore, 
the literature to which this research relates includes scholarship aimed 
at measuring quality in graduate education: 
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The most ccmpreherisi ve reviews of the literature assessing quality in 
graduate education have been conducted by Blackburn and Lingenfelter (1973) 
and by Lawrence i.nd Green (1980); Blackburn and Lingenfelter devoted their 
analysis entirely to doctoral education; Lawrence and Green reviewed the 
literature related to quality in graduate education, professional education, 
and underqraduate educatioii. 
Qual i ty Measures 

Many approaches are used to assess quality in graduate 
education. Attempts have been made to measure quality by reputation, by 
scholarly productivity, by seeking relationships between reputation and 
scholarly productivity, and by employing multiple measures. Thes' 
approaches do not represent a rigid classification scheme; it is quite 
possible that a particular study will use more than one method. Instead, 
they form a general framework to aid in understanding how quality assess- 
ments in graduate education can be differentiated. 

A number Of studies have used reputational ratings to attempt to 
measure the quality of a broad range of arademic areas in graduate education 
Hughes, 1925, 1934; Keniston, 1959; Cartter, 1966, 1977; Magoun, 1966; 
Ewell, 1966; National Science Foundation, 1969; Roose and Andersen, 1970; 
Petrowski,' Brown, and Duffy, 1973 ; Margulies and Blau, 1973 ; Blau and 
Margulies, 1974-1975; and Morgan, Kearney, and Regens, 1976. Fewer have 
sought to judge the quality of a particular field Of study. Among the 
fields that have been studied are political science (SOmit and Tanenhaus, 
1964 ), library science (Carpenter and Camenter, 1970), educational ad = 
ministration (Gregg and Sims, 1972), business administration ("The Top 15," 
1975), law ("The Popular Vote," 1976), and medicine (Cole and Lipton; 1977). 
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Seholarly proddetivlty is held by some to be an objective way to 
attempt the assessrnent of quality in graduate education. Typically these 
studies seek to measure faculty output by devising produc:ti vi ty indices 
based on scholarly publ ^'cations , honors, and p>^esentations . Clark, 1957 ; 
Grahe, 1955; Pelz and Andrews, 1956; Bayer and Folger, 1956; Eole and Eole, 
1967; Creager, 1 567 ; Myers, 1970; Stall i'ngs and Singhall, 1971; Jauch and 
Glueck, 1975; and Eox and Eatt, 1977 use scholarly productivity indices lo 
measure quality. Some studies (Perkins and Snell, 1952; Bowker, 1965; 
Jordan, 1963; and Eells, 1960) used productivity indices based on /ariables 
other than faculty output; typical variables are faculty size and degrees 
produced . 

Yet another attempt to measure quality in graduate programs i;) by 
seeking relationship?, between ept/'tation and scholarly productivity. 
Berelson, 1966; Earter, 1966; Lewis; 1958; Wispe, 1969; Knudsen, 1969; 
Erane, 1970; Shichor, 1970; Hagstrom, 19/1; Elton and Rose, 1972; Blackburn 
and Lingenfelter, 1973; Drew, 1975; Hartnett, Elark, and Bai.-d, 1976; 
Adams and Krislov, 1978; Guba and Elark, 1978 ; and Muffo, 1979 are examples. 

A departure from most of the efforts to assess quality can be found in 
the work Mary So Elark and her associates (ig74a, i974b, ig7Ba, ig75b, 
1976c, and 1977). They have used the "multiple measures" approach and 
believe that judgments of qual'ty based on many indicators have four 
advantages: l) multiple measures are more fair, 2) multiple indicators 
are more useful, 3) multidimensional procedure?, reduce the "halo" problem, 
and 4) multidimensional assessments focus on process (1976a): 
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The research described here is a multiple measures approach. It 
involves using ratings of internal processes in this case doctoral 
examinations -~ to generate measuies of qualicy in graduate pronrams. 

In 1976, Paul Dressel wrote, "It is sound practice to require that 
the dean or member of graduate council sit in on oral examinations, thereby 
placing both the student and the faculty on warning that this is more than 
a social occasion" (Dresse^ , 1975 , p. 322). Eii.^ecided in his assertion is 
an academic policy designed to maintain a level of performance on the^.^ 
examinations. The policy is essentially a quality control device, and that 
is its value. Where such a policy exists, then, the dean or faculty 
representatives of the dean's office art in a position to judge one aspect 
of graduate education. 
The Graduate School Representative 

That policy has existed in the Graduate och( i at Ohio State since 
1911 and h^s led to the generation of valuable information about doctoral 
examinations. The procedures for implementing the policy have changed over 
the last seventy years, but the philosophy has remained intact. Initially, 
the entire body of twelve Graduate Council members joined each student's 
dissertation committee and were full voting members at the oral defense of 
every dissertation. In the twenties the procedure was mcdifiec to require 
that only three Graduate Council members be present. Currently for each 
examining committee, the Dean of the Graduate School appoints a Gradtiate 
School Representative authorized to advise doctoral students to serve on 
each doctoral examination for the General Examination known elsewhere 
as the Ph.D. candidacy, lualifyihg, or preliminary examination and for 
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the Final Oral Examination cdrhmdnly known as the defense of the disserta- 
ticn: The Graduate School Representative, like the other examining com- 
mittee members, votes on the outcome of the examination and, in addition, 
reports to the Graduate School Dean about the fairness of tne examination, 
the quality of the examination, and the quality of the student's performance. 

The Graduate School logs information about each doctoral examinotion. 
The log contains the type of examination, the date of examination; the 
names and fields of the student, the adviser, and the Graduate School 
Representatives the outcome of the examination; the student's cu™lative 
point hour ratio and number of credit hours earned; and the Graduate School 
Representative's evaluation of the examination. It is possible to analyze 
one or any combinatifr'-^ of the data categories logged. How the Graduate 
School Representatives' evaluations ar: analyzed used will be described. 

When programs offering graduate ceg''ces begin program review, their 
graduate studies committee receives stcHiaard sets of descriptive informa- 
tion a'^out that graiuate program, inc"'iding dat^i about doctoral examinations. 
The doctoral examination data include tabular displays of the Graduate 
SfihQo! Reprev.entati ves ' evali-^tir ',c copies of th^ ovaluaiion inrw-, 
comolete with written r(^niments: The tcjles are prepared by type of 
examinatiofi . 

The GenL Examination Evalr.Uidn Forms yield cuantitative and qj^-^li- 
t.nlve data. The quantitative data are oerived from the three items each 
Graduate S'>.:1 Representative rates from one to seven (poor to excellent) 
0^- a Likert ^cale. The items are : 1) The a^ft roiariateness of the oral 
q .-sLions, 2) The level of difficulty of the ot^al questions, and 5) Fiow do 
V M rat: the candidate's per formance ? (the italicized w^^ appear in 
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FigUres 1, 3, 5, and 7 and refer to the eorrespondihg items). From the 
ratings quantitative data auout perceived quality arc- generated. Qualita- 
tive data dr<5 derived from written comuients the Graduate School Represents^ 
tives make: 

The rated items are expressed as mean ratings and are displayed to 
show a program's ratings in the context of the mean ratings for its College 
and for the University. These ratings are presented in tabular form as 
seen in Figurr. ' Figure 1 is the General Examination Table fur a part icu- 
lar degree program. Program ratings are taken out of the College ratings 
and College ratings are subtracted from University ratings. 

(Pigare 1 here) 

The Final Oral Examination forms provide quantitative iand qualitative 
data; The quantitative data are derived from the seven items each Gradu... 
School Representative rates from one to seven (poor to excellent) on a 
Likcrt scale. The items are: 1) The candidate's dissertation snb-Um, 
2) Soundness of the research appi'oach and application o^ research tech- 
•li.iues ( method ) , 3) Organization and style of the dissertation jdocument) , 
/I) The student's defen_se of dissertation, 5) The appropriateness of the 
oral questio-n^, 6) The level of difficulty of the O*- 1 examinations, 
7) Your g^-,eral impression of the candidate's competence (the italicized 
words appear in Figures 2, 4, and 6 and refer to the corresponding items). 
Seven items app.^r-' - en this form because the Final Oral Examination is 
qualitatively different from the General Examination. From the ratings 
quantitative data about perceived quality are gotierated. Qualitative data 
firs derived from written comments the Graduate School Representatives make. 




Figure 2 is the Final Oral Examination fable for a particular degree 
program. For the Final Oral Exanii nation, too, program ratings ar^ taken 
out of the College ratings and College scares a"3 subtracted from University 
ratings. 

(Figure 2 here) 

Un-iversity Perspective : 

Figures 1 and 2 represent the quantification of perceived quality for 
doctoral examinations in partifjlar degree programs and are presented in 
the context of their Colleges a^-' the University. Graduate School Repre- 
sentatives* evaluations can be suL :cted to even finer levels of u alysis 
within programs, hater. ;ntraprogram possibilities will be considered. 
But first, mean ratings for doctoral exam^!nations will be viewed from a 
wider angle. 

Figures 3 and 4 display the mean ratings for tpe General and Final 
Oral Examinations by outcome; pass or fail. The Graduate School Repre- 
sentatives (M) perceive differences between the passed and failed examina- 
tions. Aralysis of variance using repeated measures was conducted for 
both examinations. The F ratos were significant beyond the .01 Icvel: 

The performance of studet^.s who passed these examinations v/as rat^d 
drafnatical ly higher than those who failed (see figure 3, i teni 3 and 
figure 4, items 4 and 7) The difference in the mean ratings for the^^-- 
iteins is from 2:6 to 3.4 point:. The difference in mean ratings for items 
about the committee (see figure 3, items 1 and 2 and figure 4, items 5 and 
6) range from no difference to .6. For items inv ,ving both the ^.tudent 
and the committee (see figure 4, items 1, 2, and 3), the difference in 
mean ratings is from 1.2 to 2.0. These data show that Graduate School 
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Represeriti fives observe ciDmnldrl starrdards bei.ig applied to examinations by 
examining eemmittees regardless of wH?ther students pass or fail. They 
also perceive a large difference in student performance. 

(Figures 3 and 4 here) 

Figures 5 and 6 compare the mean ratings for the General and Finil 
Qral Examinations by Graduate School Representatives from inside and from 
outside of the students' colleges. A Graduate School Representative cai:not 
be appointed from the student's program but only from a program inside or 
outside a student's college. For example, an English professor may not 
serve as a Graduate School Representative on an English examination but 
may serve on a classics examination inside the college or on a physics 
exaniination outside the college. 

Again, analysis of variance using repeated measures was conducted for 
both examinations. The F ratio was not significant at the :61 level in 
either ease. Thus, we found no evidence of a significant difference between 
how Graduate School Repr^esentati ves rate examinations conducted inside the^r 
own colleges compared to their ratings of examinations oatside their own 
colleges. These results suggest that faculty members have some comn^on 
perceptions about the quality of doctoral examinations, regardless of 
discipline: Viewed from this perspective, notions like "paradigm develop- 
ment" and the "halo effect" may not operate as strongly as some would have 
us bel ieve . 

(Figures 5 and 6 

Intraprogram Perspective 

It is possible to present intraprogram configurations ct Graduate 
School Representatives' evaluations. Figure 7 shows a program with two 

er|c 1:2 



major fields of specialization in its graduate program: Field 1 and 
Field 2: The faculty in this program wanted to compare the Graduate School 
Representative- ' evaluations in the two Fields. Ratings for each Field 
are shown In Figure 7. 

(Figure 7 here) 

Some faculty in the program expected mean ratings for Field 2 would 
be much lower than the mean ratings for Field 1. The mean ratings for the 
two Fields show only slight differences; and the differences are not all 
in one direction. In this case, the evidence helped to expose myth and to 
enhance the stature of one group of faculty .hj^bers within the program. 

In another case a program chairman submitted a proposal to the Graduate 
School requesting that a new field of specialization be added to the 
graduate program. At the time, 'the program had two fields of specialization. 
Sne new faculty position was requested for the proposed field. The remain- 
ing faculty would come entirely from one of the other two fields. In con- 
sidering the program's proposal, the graduate School analyzed the Graduate 
School Representatives' evaluations for the program and for the faculty 
group designated in the proposal. The mean ratings "for the faculty group, 
the program, the Eollege, and the University were compared. The program 
was rated lower than the College and the faculty group was rated lower 
than the program. 

To this portrait, other Information was added. Data comparing the 
distribution of advisees for the faculty group, the program, the College, 
and the University were analyzed. These data showed that the faculty group 



had high advisee loads. A profile of the faculty group emerged shov/ing 
them to have low Graduate School Representatives' evaluations and to have 
high advisee loads. The profile was given to the prdgraiti chairman. 

The response indicates tha c the data had ah influence. First of all, 
the faculty remained committed to pursuing the new field of specialization 
Secondly, they decided to close admissions for both doctoral and masters 
applicants in the field of specialization in which the faculty group teach 
This was done, they said, to bring the advisee load per faculty member in 
that field more closely in line with their Gollege and with the University 
Finally^ they stipulated that the new faculty member requested would be 
assigned advisees from the new field of specialization only. 
eODA 

We have described how qualitative judgments about doctoral examina- 
tions in the Graduate School at Ohio State are quantified and have 
shown ways to array and use these data. The Graduate School Representa- 
tives' evaluations are internal indicators of quality; but they are some- 
what different from the internal indicators described by Glark (1977). 
Clark discusses internal indicators of quality in the context of judgments 
and perceptions of students, faculty, and alumni from within a program 
and about that same program. Graduate School Representatives' evaluations 
are internal indicators of quality generated from within the University by 
faculty members who are outside the programs of the examinations they 
judge: Another difference between the two types of Internal indicators 
concerns comparative data. Glarfc's indicators focus solely on individual 
programs where the Graduate School Representatives' evaluations show a 
program in relation to its College and the University. 
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Have the Graduate School Representatives' evaluations been useful? 
The answer is: they have: Program faculty have changed thei. examination 
procedures as a result of the ratings and the comments made by Graduate 
School Representatives. Some have changed the substance of their doctoral 
examinabions. We noted above that the ratings have influenced changes in 
percepcions among program faculty and that they have been partly responsible 
for the suspension of graduate admissions in a field of specialization. 
Finally, many faculty have said that the evaluations give them one more 
way of knowing about thair programs, which in itself is an occasion for 
program improvement. 

Like Clarke we believe there is value in using multiple measures U 
assess quality in graduate education. The Graduate School Representatives' 
evaluations are added to other internal and external quality measures to 
help programs identify their strengths and weaknesses: 0nce strengths 
and weaknesses are identified, program improvement is possible. 

Programs at Ohio State have the opportunity to consider and to reflect 
upon multiple quality measures and a variety of other data during program 
review. Information from a program's students, faculty, and alumni and 
input from the Qffice of Academic Affairs, the Graduate School, and the 
College office are considered in the Self-Study phase of review. Experts 
from outside the University conduct a study of the program in the External 
Review phase of program review. The sole aim of program review is to 
improve programs. Therefore, the information collected before and during 
review forms the basis for a searching examination which can lead to 
improvement. 

ERIC 
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Quality is hot an absolute and cannot be expressed in absolute terms. 
There is ho ruler to measure quality in graduate education. Therefore, it 
is important that institutions establish many devices for determining how 
well the'.r programs perform. Having faculty members who are experts in 
conducting doctoral examinations judge doctoral examinc.tions is one device 
that works at Ohio State: It may work elsewhere. Regardless, where the 
assessment of quality is concerned, more measures are better than one! 
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FIGURE 1 

Genera! Examinations 
Mean Rating of Graduate School Representatives* Evaluations 
(Autumn 1976-Spnng 1980) 
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_ FIGURE 2 
Final Oral hxattiihations 
Mean Rating of Graduate School Representatives' Evaluations 
(Autumn 1 976-Sprmg lOJBOj 



i 




1 terns 
















A~ 1 irk r~i r? t ; r~\ — t 

iwl tici 1 1 ivci L i n J 


r r (. > t ) » L . 1 1 4 




Document 


Defen se 


Questions 


Difficulty 


CoiTif^etenoi 




1 

_i_ 


2 


3 




5 


6 


7 


















- 

7. 0 
















, 8 
































. 2 


1 














6. 0 
















. 8 
















. 6 










OSU /Prog 




— 
Prog 
















0StJ 


. 2 


OSU /Prog 


OSU /Prog 




/roll / 
Prog 


_ _ . _ 
Coll 


OSU/CdII 


Coll 


5. 0 


Coll 




OSU 






- 




u 
• (J 






Prog 






Prog 




• \j 






Coll 










n 
















. 2 
















4.0 
















.8 
















.6 
















.H 
















. 2 
















3.0 
















.8 
















.6 
















.4 
















.2 
















' z.O 

i 

















U nit 



jram 
College 
OSU 



Number of Evaluations 
45 
23 
1697 



DD /JC /wp 
a/21/8i 



erJc 



FIGURE 3 

Cenerai Exaitjindtidns 
Mean Rating of Graduate S_cl1obi_ Rejoresehtativcs' tvaluatidhs 
(Autumn 1 976 -Spring 
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FIGURE it 

Final Oral Exartiihatibhs 
Mean Ratihcj of Graduate School Repiresentatives' Evaluations 
(Autumn 1975-Sprihg 1986) 
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FIGURE 5 
General Examinatidhs 
Mean Rating of Graduate Sc_hobi_ Representatives" Evaiuatibhs 
(AutUrrih i976-Sprihg 1980) 
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FIGURE 6 
Fiha' Oral Examihatibhs 
Mean Rating of Graduate Schbd? Representatives' Evaluations 
(Autumn 1976--Spring 1980) 
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FjcURE 7 

... _ _ General Examinations 
Menn Rating of Graduate School Representatives' h valijatioris 
(Autumn 1 976-Spring 1980} 
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